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Abstract 

The  goal  of  this  research  is  to  suggest  a  framework  for  developing  measures  of  success 
for  corporate  level  Air  Force  acquisition  initiatives.  Because  this  research  is  exploratory,  it 
focuses  on  only  one  initiative:  the  2002  initiative  “Focus  on  results,  not  process.”  A  qualitative 
method  approach  was  used  to  suggest  a  four  part  framework.  Through  the  review  of  literature, 
common  steps  for  creating  metrics  were  established  and  recurrent  characteristics  of  good 
metrics  were  identified.  Then  interviews  were  conducted  with  acquisition  practitioners  who  have 
experience  with  the  initiative.  Finally,  those  three  parts  were  applied  to  the  initiative  as  a  case 
study  and  metrics  suggested  as  a  result. 

This  study  gives  Air  Force  leaders  clear,  implementable  metrics  that  can  be  used  as 
measures  of  success  for  the  initiative,  and  provides  recommendations  to  improve  this  initiative’s 
performance  and  that  of  future  corporate  Air  Force  acquisition  initiatives.  This  study  also  gives 
leaders  insight  into  whether  or  not  this  initiative  and  others  like  it  are  an  appropriate  and 
effective  way  to  drive  the  changes  they  are  meant  to  bring  about.  Finally,  from  a  broader 
perspective,  the  framework  used  in  this  study  can  be  used  to  develop  metrics  for  other 
corporate  level  initiatives. 

Introduction 

Almost  since  its  inception  in  1947,  the  Air  Force  has  sought  to  reform  the  way  it  procures 
weapon  systems.  Many  factors  involved  in  the  weapon  system  acquisition  process  are  external 
to  the  Air  Force  and  out  of  its  direct  control  (i.e.,  Congressional  constraints,  the  pace  of 
technology  development,  constantly  changing  world  situations).  However,  self-imposed 
administrative  hurdles  are  an  internal  factor  that  the  Air  Force  can  change  in  order  to  help 
improve  its  procurement  practices. 

To  target  the  elements  of  the  acquisition  process  within  its  control,  the  Air  Force  began 
implementing  a  series  of  acquisition  reform  initiatives  in  1995.  These  initiatives,  referred  to  as 
“Lightning  Bolts,”  were  created  in  direct  response  to  Air  Force  leadership’s  growing  concerns 
that  it  takes  too  long  to  put  weapon  systems  in  the  hands  of  the  warfighters  (Department  of  the 
Air  Force,  2003).  Collectively,  their  purpose  was  to  serve  as  the  catalyst  by  which 
administrative  changes  are  made  in  Air  Force  business  practices  (Senate  Armed  Services 
Committee,  2002).  However,  little  is  known  about  how  to  gauge  the  success  of  these  initiatives. 
Many  metrics  have  been  suggested  for  gauging  the  success  of  acquisition  reform  attempts 
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within  the  Department  of  Defense  (DoD),  but  most  of  the  metrics  remain  slated  for  use  only  in 
individual  acquisition  program  offices  (Pope,  1997).  No  list  of  standard  metrics  exists,  and  there 
are  no  generally  applicable  and  logical  methods  to  measure  the  performance  of  acquisition 
reform  initiatives  today  (Beamon,  1999;  Pope,  1997).  Accordingly,  this  project  is  designed  to 
help  better  understand  these  initiatives  and  how  to  establish  acquisition  based  measures  of 
success. 

Literature  Review 

This  section  provides  the  foundation  for  the  study  by  addressing  several  relevant 
literature  streams.  First,  the  background  and  purpose  of  the  acquisition  reform  initiatives  are 
reviewed.  Next,  metrics  are  discussed,  including  general  steps  involved  in  developing  metrics, 
attributes  of  good  metrics,  and  how  metrics  can  be  applied  to  acquisition  reform  initiatives.  This 
review  identifies  commonalities  among  theories  of  metric  development  that  highlight  common 
attributes  of  good  metrics.  Finally,  a  list  of  metrics,  generated  through  a  series  of  interviews,  will 
be  assessed  against  the  characteristics  of  good  metrics  to  construct  a  set  of  useable  metrics  for 
the  2002  initiative  entitled  “Focus  on  results,  not  process.” 


Acquisition  Reform  Initiatives 

On  27  February  2002,  in  an  update  to  the  Senate  Armed  Services  Committee  (2002)  on 
the  Air  Force’s  on-going  acquisition  reform  efforts  and  progress,  the  Assistant  Secretary  of  the 
Air  Force  for  Acquisition,  Dr.  Marvin  Sambur,  reaffirmed  the  goal  set  for  the  Air  Force  by  the 
President  and  the  Secretary  of  Defense  to  transform  the  military  and  improve  how  it  does 
business;  specifically,  the  Air  Force  must  reduce  cycle  times,  improve  its  ability  to  estimate  both 
costs  and  schedules,  increase  delivery  speeds,  and  generally  work  to  regain  credibility  with  the 
war  fighter. 

In  an  effort  to  address  senior  leaders’  desire  to  improve  speed  and  credibility,  acquisition 
leaders  released  six  acquisition  reform  initiatives  in  2002;  similar  sets  of  initiatives  had  been 
released  in  groups  of  six  to  ten,  approximately  every  two  years  since  1 995.  Two  of  the  six 
initiatives  released  in  2002  were  process  oriented.  One  initiative,  entitled  “Focus  on  results,  not 
process”  encouraged  streamlining  existing  acquisition  processes,  challenging  those  that  do  not 
add  value,  and  getting  rid  of  the  processes  that  do  not  make  sense.  The  second  process 
initiative  was  designed  to  strengthen  continuing  process  improvements  and  communication 
between  the  government  and  contractors  by  creating  a  knowledge  pipeline.  (Druyun,  2001; 
Senate  Armed  Services  Committee,  2002) 

The  other  four  2002  initiatives  are  people-oriented.  These  four  initiatives  sought  to 
encourage  cooperation  between  warfighters  and  acquisition  practitioners  during  the 
development  and  incremental  delivery  of  warfighting  capabilities;  to  give  managers  a  single 
point  of  contact,  the  Acquisition  Center  of  Excellence  (ACE)  office  to  help  them  remove 
administrative  and  bureaucratic  stumbling  blocks,  thus  freeing  them  to  be  innovative;  to  change 
the  ingrained  culture  of  the  acquisition  workforce  toward  a  bias  for  innovation;  and  to  encourage 
leveraging  the  Air  Force’s  buying  power  in  services  through  the  creation  of  a  Program  Executive 
Officer  (PEO)  for  service  contracts. 

This  research  will  focus  on  the  initiative  arguably  most  closely  linked  with  the  corporate 
goal  set  for  the  Air  Force  to  improve  speed  and  credibility:  “Focus  on  results,  not  process”. 

This  initiative  was  designed  to  drive  a  “clean-sheet”  approach  to  acquisitions  by  streamlining 
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processes  in  order  to  remove  non  value-added  steps  (Senate  Armed  Services  Committee, 
2002).  In  other  words,  the  initiative  sought  to  free  up  the  administrative  hands  of  acquisition 
practitioners  to  allow  them  to  be  as  innovative  as  possible  within  the  confines  of  the  law.  How 
well  this  and  other  initiatives  are  achieving  their  desired  goal  is  an  open  question,  and  without 
valid  metrics,  it  will  remain  so. 


Metrics 

The  Metrics  Handbook  developed  by  the  then  Air  Force  Systems  Command  (1991) 
defines  metrics  as  meaningful  measures  that  allow  action  to  be  taken.  Similarly,  Antanitus 
(2003)  calls  metrics  items  you  would  like  to  measure.  Metrics  emphasize  the  customer,  support 
organizational  objectives  and  goals,  facilitate  process  understanding,  and  encourage  continual 
improvement  of  how  business  is  done  (AFSC,  1991). 

Metrics  improve  performance  (Antanitus,  2003;  Buchheim,  2000;  Rummler  and  Brache, 
1995)  by  indicating  how  well  an  organization  is  performing  (Goett,  2003;  Klapper,  Hamblin, 
Hutchison,  Novak,  and  Vivar,  1999;  Lambert  and  Pohlen,  2001;  Milliken,  2001).  Metrics  not 
only  examine  how  an  organization  is  performing  (Milliken  2001),  but  more  importantly,  should 
help  it  perform  better  (Hammer,  2001). 

Metrics  are  used  to  improve  performance  and  properly  structured  metrics  can  drive 
superior  performance.  Keebler  et  al.  (1999)  discovered  a  great  disparity  in  levels  of 
organizational  performance  and  found  the  most  important  factor  driving  superior  performance 
among  the  organizations  in  their  study  was  the  presence  of  well-utilized  and  properly  structured 
measurement  programs.  Inadequately  structured  metrics,  on  the  other  hand,  can  drive  the 
wrong  behaviors  and  even  result  in  dysfunctional  behaviors  (Neely,  Richards,  Mills,  Platts,  and 
Bourne,  1997). 

Additional  evidence  of  how  the  use  of  metrics  has  been  shown  to  improve  performance 
is  seen  in  the  literature  on  goal  setting  (see,  for  example,  Latham  and  Yukl,  1975;  Latham  and 
Locke  1979;  Locke  1968;  Locke,  Shaw,  Saari,  and  Latham,  1981;  Locke  1982).  a  significant 
amount  of  data  attests  to  the  presence  of  increased  performance  when  goals  are  set,  and, 
therefore,  when  metrics  are  used.  Within  the  context  of  goal-setting,  metrics  are  the  feedback 
mechanism  by  which  progress  toward  organizational  goals  is  measured  (Office  of  the  Linder 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics,  1999).  For  example,  the 
Commander’s  Initial  Guidance  on  improving  speed  and  credibility  states  that  the  overall  goal  is 
to  shorten  the  time  it  takes  for  decisions  and  getting  more  capable  weapon  systems  out  to  the 
warfighter  by  a  factor  of  four  (Department  of  the  Air  Force,  2003).  Correspondingly,  the  metric 
to  determine  if  that  goal  is  met  will  be  cycle  time. 

In  addition  to  metrics’  importance  because  of  improved  performance,  Keebler  et  al. 
(1999)  point  out  that  measures  aid  companies  in  determining  how  to  remain  competitive  and 
confirm  the  value  customers  place  on  their  services.  And  the  underlying  truth  within  the  axiom 
that  what  gets  measured  gets  attention  is  yet  another  reason  to  use  metrics  (Eccles,  1991 ; 
Osborne  and  Gaebler,  1992). 

If  metrics  are  important,  what  makes  a  good  metric?  According  to  Clark  and 
Wheelwright  (1994:262),  there  are  two  types  of  measures:  results  measures,  which  tell  a  team 
where  it  currently  stands  in  its  attempt  to  reach  a  goal,  rather  than  how  it  got  there  or  what  it 
could  do  differently;  and  process  measures  which  look  at  activities  and  tasks  within  an 
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organization  that  produce  given  results.  Also,  metrics  can  be  expressed  both  qualitatively  and 
quantitatively  (Beamon,  1999).  Quantitative  metrics  are  frequently  preferred  because  qualitative 
metrics,  like  “poor,”  “fair,”  and  “good,”  are  vague  and  hard  to  use  in  a  meaningful  way  (Beamon, 
1999).  However,  quantitative  metrics  may  not  adequately  discuss  a  system’s  performance  and, 
as  a  result,  may  be  just  as  vague  (Beamon,  1999).  It  should  not  be  assumed  that  specific 
quantitative  goals,  and,  in  turn,  metrics,  are  inevitably  beneficial;  some  areas  where  results  are 
more  difficult  to  measure  may  require  qualitative  goals,  and,  in  turn,  qualitative  metrics  (Locke, 
1978).  The  decision  between  qualitative  and  quantitative  metrics  depends  upon  the  nature  of 
the  system  for  which  the  metrics,  or  goals,  are  being  established. 

Steps  to  Create  Metrics. 

By  comparing  the  numerous  methods  for  systematically  developing  metrics  that  exist 
within  the  literature,  this  research  found  that  nearly  all  of  the  methods  share  three  common 
steps  which  will  later  be  discussed.  Of  the  literature  reviewed,  twelve  authors  presented 
thirteen  general  frameworks  for  creating  metrics  (AFSC,  1991;  Antanitus,  2003;  Brown,  1996; 
Buckheim,  2000;  Clark  and  Wheelwright,  1994;  Eccles  and  Pyburn,  1992;  Evans  and  Lindsay 
2002;  INCOSE,  1998;  Keebler  et  al. ,  1999;  Mentzer  and  Konrad1991;  Pinker,  Smith,  and 
Booher,  1997;  Rummlerand  Brache,  1995).  The  number  of  steps  involved  in  each  framework 
ranged  from  three  steps  up  to  eleven  steps.  For  example,  Clark  and  Wheelwright  (1994) 
suggest  a  four  step  method:  a)  define  factors  critical  to  customer  satisfaction;  b)  map  cross¬ 
functional  process  through  which  results  are  obtained;  c)  identify  capabilities  and  tasks 
necessary  to  complete  process  successfully;  and  d)  design  measures  to  track  those  capabilities 
and  tasks.  Rummler  and  Brache  (1995)  recommend  a  similar  four  step  sequence:  a)  clearly 
establish  the  most  important  outputs  of  the  process,  job,  or  organization;  b)  for  each  output, 
establish  the  “critical  dimensions”  of  performance;  c)  create  measures  for  every  critical 
dimension;  and  d)  create  standards,  or  goals,  for  each  measure.  In  contrast,  Eccles  and  Pyburn 
(1992)  suggest  a  five  step  process  that  does  not  share  the  three  steps  found  to  be  common 
among  the  other  authors:  a)  choose  non-financial  measures  that  will  compliment  financial 
measures,  determine  relationships  between  them,  and  create  firm’s  business  performance 
model;  b)  establish  methodology  to  be  used  to  take  the  measures;  c)  select  the  frequency  and 
layout  of  performance  measurement  reports;  d)  adjust  how  personnel  are  compensated  and 
evaluated  to  encourage  desired  behavioral  changes  that  will  improve  activity  performance;  and 
e)  realize  a  key  element  of  performance  measurement  system  is  that  it  will  evolve  with  time  as 
managers  grow  and  increase  their  knowledge  of  measures’  relationships  to  one  another  and  as 
conditions  change. 

No  empirical  evidence  was  found  within  the  literature  to  suggest  that  any  one  particular 
method  was  better  to  use  than  any  other.  Many  differences  exist  among  the  authors’ 
approaches,  but  three  basic  steps  remained  common  among  eleven  of  the  thirteen  frameworks 
examined  (INCOSE,  1998).  First,  establish  a  starting  point  upon  which  to  base  the  metrics; 
determine  what  you  want  to  measure.  Second,  identify  the  most  important  elements  of  what 
you  want  to  measure.  Third,  create  specific  metrics  for  those  critical  elements  so  as  to  improve 
the  performance  of  the  item  being  measured.  If  metrics  are  created  by  systematically  following 
these  three  general  steps  and  they  possess  the  attributes  of  good  metrics  they  will  be  properly- 
structured  metrics  and  will  have  the  potential  to  drive  superior  performance  (Keebler  and  others, 
1999). 
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Attributes  of  Good  Metrics. 

Certain  characteristics  distinguish  good  metrics  from  bad  ones  and  well-designed 
metrics  possess  those  good  characteristics.  Fourteen  authors  in  the  literature  describe  forty- 
three  distinct  attributes  that  good  metrics  possess  (AFSC,  1991;  Antanitus,  2003;  Beamon, 

1999;  Brown,  1996;  Buckheim,  2000;  Cohen  2003;  Evans  and  Lindsay  2002;  INCOSE,  1998; 
Kaplan  1991;  Keebler  et  al. ,  1999;  Mentzerand  Konrad1991;  Milliken,  2001;  Pinker,  Smith,  and 
Booher,  1997;  Rummler  and  Brache,  1995).  Beamon  (1999),  for  example,  says  that  good 
metrics  have  six  characteristics:  consistency  with  organizational  goals,  inclusiveness  of 
pertinent  aspects,  measurability,  meeting  of  customer  goals  and  values,  relate  to  strategic  goals 
and  mission  of  organization,  and  universality.  In  comparison,  Buchheim  (2000)  describes  good 
metrics  as  having  eight  characteristics,  only  one  of  which  is  common  with  those  cited  by 
Beamon  (i.e,  relating  to  strategic  goals  and  mission).  According  to  Buchheim  (2000),  good 
metrics:  have  a  defined  sensor  that  gathers  and  records  data,  like  an  automated  test  station 
data  file  or  a  clerk;  have  a  defined  unit  of  measurement  (e.g.,  hours  per  widget  produced);  are 
meaningful  to  the  customer;  measure  results  versus  process  (e.g.,  measure  the  level  of  skill 
demonstrated  using  a  widget  versus  the  number  of  days  spent  attending  training  sessions); 
have  a  regular  frequency  with  which  reports  and  measurements  are  done  (e.g.,  monthly 
average  failure  rate);  are  simple  to  use;  and  are  understandable.  Evans  and  Lindsay  (2002) 
agree  with  both  Beamon  and  Buchheim  that  good  metrics  relate  to  the  strategic  goals  and 
mission  of  the  organization  involved,  but  also  state  that  good  metrics  are  actionable  and  useful. 

Summarizing  the  commonalities  among  the  various  lists  of  attributes,  six  authors  claim 
metrics  should  relate  to  the  organizational  mission  and  strategic  goals,  five  suggest  simplicity  is 
an  important  quality  of  metrics,  and  five  state  good  metrics  are  meaningful  to  customers.  Four 
authors  point  out  metrics  should  be  understandable  and  derivable  from  economically  collectible 
data  (i.e.,  cost  effective).  All  other  attributes  are  common  among  three  authors  or  less. 

Application  of  Metrics  to  Acquisition  Reform  Initiatives. 

The  military  acquisition  community  manages  and  oversees  the  activities  involved  in  the 
procurement  of  weapon  systems,  from  initial  development  and  procurement,  through  delivery  to 
the  war  fighters  and  to  the  end  of  a  weapon  system’s  life  cycle  when  it  is  retired.  This 
comprehensive  system  suggests  a  supply  chain  perspective  is  appropriate  for  analyzing  the 
weapon  system  acquisition  and  management  process  (Klapper  et  al.,  1999;  Monczka  et  al., 
2004). 

Metrics  appropriate  for  acquisition  reform  enable  an  organization  to  assess  reform 
initiatives’  effectiveness  and  implementation  on  both  acquisition  programs  and  the  acquisition 
reform  process  itself  (Pope,  1997).  Groups  within  the  DoD  have  proposed  various  metrics  to 
measure  acquisition  reform,  but  most  metrics  have  been  specific  to  individual  acquisition 
programs  (Pope,  1997)  and  no  systematic  approach  to  performance  measurement  or 
standardized  set  of  metrics  for  acquisition  reform  initiatives  exists  (Beamon,  1999;  Pope,  1997). 

In  an  effort  to  address  the  lack  of  standardized  metrics  for  acquisition  initiatives,  the 
Acquisition  Reform  Benchmarking  Group  (ARBG)  was  established  by  the  DoD  in  1996  to  help 
measure  progress  within  the  arena  of  acquisition  reform  (Pope,  1997).  Pope  (1997)  determined 
that  the  findings  of  the  ARBG  divide  metrics  into  three  levels:  program,  subordinate,  and 
enterprise.  Metrics  at  their  most  basic  level  measure  elements  within  individual  acquisition 
programs  (DiCicco,  2003).  Subordinate  metrics  measure  factors  that  feed  into  the  highest  level 
of  metrics,  which  are  enterprise  metrics.  Enterprise  metrics  measure  the  efficiency  of 
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overarching  or  generalizable  processes  that  should  be  measured  across  the  whole  Air  Force 
(DiCicco,  2003;  Pope,  1997).  Enterprise  metrics  include  cost,  schedule,  performance,  and 
training  metrics.  The  acquisition  initiative  this  research  focuses  on  pertains  to  enterprise-level 
acquisition  and  the  metrics  this  research  will  recommend  be  used  to  assess  that  initiative  are 
enterprise  level  metrics. 

Pope  (1997)  found  that  metrics  can  also  be  categorized  by  the  three  types  of  activities 
that  they  measure,  as  defined  by  the  1995  Process  Action  Team  (PAT)  for  contract 
administration  reform:  go/no-go,  activities,  and  behavioral  changes.  Go/no-go  metrics  show 
whether  or  not  an  activity  has  taken  place.  Activity  metrics  illustrate  how  extensively  an  action 
is  occurring.  And  behavioral  change  metrics  assess  whether  actions  are  creating  the  desired 
change  in  behavior  or  results.  This  research  seeks  in  large  part  to  determine  whether  or  not  the 
use  of  the  acquisition  initiative  of  interest  is  an  effective  way  to  bring  about  the  desired  changes 
in  the  acquisition  practitioners’  behavior. 

Methodology 

Based  on  the  nature  of  the  research  question,  a  qualitative  approach  was  used  to  guide 
the  research  project  based  on  the  procedures  outlined  by  Creswell  (2003).  Data  was  collected 
through  a  series  of  semi-structured  interviews.  After  the  interviews  were  transcribed,  the 
transcriptions  were  broken  down  into  statements  and  analyzed  for  common  themes. 

Specifically,  the  interviews  were  designed  to  generate  a  list  of  metrics  that  can  be  used  to 
measure  the  reform  initiative  of  interest  and  to  determine  the  extent  to  which  this  initiative  was 
facilitating  desired  changes. 

Interview  Sample. 

In  2001,  Acquisition  Centers  of  Excellence  (ACE)  were  established  for  the  Air  Force,  Air 
Force  Materiel  Command,  Air  Force  Product  Centers,  and  Air  Logistic  Centers  to  lead 
acquisition  reform  efforts  (New  Acquisition  Center  Provides  Warfighting  Capabilities,  2001; 
Lightning  Bolts,  2004).  Part  of  their  duties  is  to  oversee  the  implementation  of  the  major 
acquisition  reform  initiatives.  As  a  result,  the  ACE  offices  have  helped  system  program  office 
(SPO)1  leadership  understand  and  implement  the  initiatives.  Therefore,  in  this  research, 
members  at  the  ACE  offices  and  various  system  program  offices  (SPOs)  within  the  Air  Force’s 
Product  Centers  and  Air  Logistics  Centers  were  interviewed.  To  further  broaden  the  research 
sample,  individuals  holding  various  acquisition  related  positions  within  Air  Combat  Command, 
Air  Force  Space  Command,  and  Air  Staff  were  also  interviewed.  Modeling  Carter  and  Jennings 
(2002),  the  sample  interviewed  was  chosen  with  the  intent  of  getting  a  high  degree  of  variation 
among  managerial  levels  in  order  to  get  a  higher  range  or  scope  of  data. 

The  ACE  personnel  were  asked  to  identify  interview  participants,  within  the  SPOs,  who 
have  experience  with  the  initiative  of  interest.  Of  the  fourteen  Center  ACEs  queried  for 
assistance,  two  provided  contact  information  for  interview  participants.  The  two  respondent 
Center  ACEs  were  from  separate  locations;  participants  from  Acquisition  Category  (ACAT)  one 
and  two  programs  were  identified  at  one  location  and  from  ACAT  three  programs  at  the  other 


1  The  Air  Force  recently  renamed  its  acquisition  organizations  to  reflect  better  the  more  standard  military 
terminology  of  wings,  groups,  and  squadrons.  This  paper  retains  the  previous  terminology  of  program 
offices  and  the  like,  because  readers  are  likely  to  be  more  familiar  with  these  names. 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  FOR  INFORMED  CHANGE 

-287- 


location.  ACAT  describes  program  size  and  dollar  amount  and  ranges  from  one,  being  the 
largest  and  most  expensive  programs,  to  three,  being  the  smallest  and  least  expensive.  Six 
Center  ACEs  gave  negative  replies  (three  of  which  were  initially  non-respondent,  but  gave 
negative  replies  when  asked  again)  and  cited  several  reasons  why:  individuals  at  their  location 
had  no  experience  with  the  initiative  of  interest;  they  sent  a  message  out  to  SPOs  asking  for 
participants  and  got  no  replies  back;  due  to  the  nature  of  the  mission  at  their  location  (e.g.,  a 
test  and  evaluation  organization)  they  did  not  use  the  initiative  of  interest;  they  never  received 
the  initial  request  from  the  SAF/ACE  asking  for  assistance  with  the  research;  and  since  their 
location  was  neither  a  Product  nor  Logistics  Center  (e.g.,  a  Test  Center),  they  thought  the 
request  for  research  assistance  did  not  apply  to  them.  Some  Center  ACEs  cited  more  than  one 
reason  for  their  negative  replies.  The  six  remaining  Center  ACEs  were  completely  non- 
responsive  even  after  being  queried  a  second  time. 

The  low  Center  ACE  response  rate  indicated  a  broader  interview  sample  was  needed 
and  that  individuals  with  acquisition  experience  from  Air  Combat  Command,  Air  Force  Space 
Command,  and  Air  Staff  should  be  included  among  interview  participants.  Additional 
participants  were  identified  with  the  assistance  of  the  SAF/ACE,  through  interview  participants 
recommending  that  other  specific  individuals  be  contacted  for  interviews,  and  through  personal 
contacts  of  the  researcher.  A  total  of  twenty  five  participants  were  identified  and  interviewed, 
but  only  twenty  three  interviews  were  usable;  nineteen  oral  interviews  were  successfully 
transcribed,  two  oral  interview  recordings  were  inaudible  and  subsequently  unusable,  two 
interviews  were  recorded  using  only  notes  taken  during  the  interviews,  and  two  interviews  were 
conducted  via  email. 

Interview  Correspondence. 

Potential  interview  participants  were  identified  and  then  contacted  to  determine  their 
willingness  to  participate.  The  interview  questions  were  provided  prior  to  conducting  the 
interviews  so  that  participants  could  prepare,  in  hopes  of  making  the  interviews  more  efficient 
and  effective.  After  interview  candidates  were  identified  and  invited  to  participate,  each  was 
contacted  via  email  or  telephone  to  schedule  an  interview  time.  Prior  to  the  scheduled 
interviews,  each  participant  was  contacted  to  confirm  his  or  her  availability  for  the  interview. 
Then,  the  interviews  were  conducted  face-to-face  and  over  the  telephone  at  scheduled  times, 
and  using  email.  Notes  were  taken  during  each  verbal  interview,  and  interviews  were  recorded 
and  transcribed  (with  the  interviewee’s  permission)  whenever  possible.  All  interviewees  were 
promised  anonymity. 

Interview  Method. 

The  interviews  included  open-ended  items,  allowing  participants  to  go  in  different 
directions.  However,  in  this  research,  a  semi-structured  interview  approach  was  used  in  order 
to  address  the  topics  of  interest  about  participants’  use  of  the  focal  reform  initiative,  within  the 
interview  time  constraints.  A  semi-structured  interview  enabled  the  research  to  follow  standard 
questions  while  allowing  the  latitude  to  include  a  few  tailored  questions  to  probe  or  clarify  a 
participant’s  reasoning.  (Leedy  and  Ormrod,  2001) 

Analysis  Technique 

The  qualitative  data  collected  was  analyzed  using  Creswell’s  (1997)  data  analysis  spiral. 
Using  this  spiral,  data  was  reviewed  multiple  times  while  going  through  the  following  steps. 

First,  the  data  was  organized  using  a  computer  database,  and  broken  into  smaller  text  units 
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(i.e. ,  sentence  and  individual  words).  Second  the  data  was  perused  for  potential  themes  and 
categories.  Third,  each  individual  datum  was  classified  using  the  major  themes.  Finally,  the 
data  was  integrated  and  summarized. 

Two  major  strategies  were  employed  to  ensure  validity.  First,  peer  debriefing  of 
experienced  acquisition  practitioners  familiar  with  this  area  of  research  helped  ensure  the 
accuracy  of  the  findings  and  to  make  the  explanation  of  this  research  clearer  for  an  outside 
audience  (Creswell,  2003).  Secondly,  following  Isabella  (1990)  and  Creswell  (2003),  external 
auditors  were  used  to  review  the  entire  research  project.  As  part  of  the  qualitative  data 
analysis,  recurrent  themes  were  identified  and  interview  data  categorized  accordingly.  Non¬ 
acquisition  and  acquisition  professionals  who  were  new  to  the  research  project  categorized 
interview  statements  under  the  themes  they  thought  were  appropriate  matches.  The 
independent  categorization  provided  by  the  auditors  validated  the  primary  classification  and 
synthesis  of  the  data  by  the  research  team. 

Data  Analysis 

This  section  discusses  the  analysis  of  the  data,  which  revealed  many  patterns  and 
themes  that  address  the  overall  research  problem. 

Interview  Participants 

Interviews  were  conducted  with  officers  and  civilians  from  Air  Combat  Command,  Air 
Force  Materiel  Command,  Air  Force  Space  Command,  and  the  Air  Staff.  Individuals  ranged  in 
rank  from  GS-12s  to  Senior  Executive  Service  (SES)  members  and  General  Officers  and  held  a 
variety  of  acquisition  positions  from  System  Program  Office  (SPO)  level  program  workers,  to 
Program  Executive  Officers  (PEOs)  and  Center  Commanders,  to  staff  positions  with  the 
Assistant  Secretary  of  the  Air  Force.  Their  time  in  federal  service  ranged  from  five  years  up  to 
thirty  one  years.  The  high  degree  of  variety  in  participants’  managerial  levels  and  areas  of 
expertise  provided  a  high  range  or  scope  of  data  (Carter  and  Jennings,  2002). 

For  purposes  of  maintaining  participant  confidentiality,  GM-15s,  GS-15s,  and  Senior 
Executive  Service  members  (excluding  those  within  the  Air  Staff),  and  System  Program 
Directors,  Program  Executive  Officers  (PEOs),  and  Center  Commanders  who  participated  in  this 
research  will  be  categorized  as  “middle  management.”  Interview  participants  referred  to  this 
grouping  of  people  as  middle  management,  to  Assistant  Secretary  of  the  Air  Force  equivalent 
positions  and  above  as  “senior  leadership,”  and  to  Deputy  System  Program  Director  equivalent 
positions  and  below  as  “SPO  level  workers.”  Most  so-called  middle  managers,  and  some  so- 
called  workers,  are  actually  relatively  senior,  experienced  personnel;  the  categories  represent  a 
self-classification  by  the  participants  of  their  positions  relative  to  other  participants’  positions. 

Analysis  Overview 

Modeling  Isabella  (1990),  interview  participants’  responses  to  each  interview  question 
were  systematically  and  carefully  examined  to  identify  both  recurrent  themes  and  unique.  Every 
interview  transcript  was  reviewed  and  sections  of  the  interviews  were  excerpted  verbatim  and 
typed  on  separate  pieces  of  paper  to  illustrate  the  nucleus  of  each  individual’s  statements 
(Isabella,  1990).  After  excerpts  were  perused,  they  were  classified  into  recurrent  themes  and 
categories  (Creswell,  2003;  Isabella,  1990).  Roughly  seven  hundred  excerpts  were  recorded. 
Category  coding  accuracy  was  ensured  using  external  auditors  (Creswell,  2003;  Isabella,  1990). 
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Reviewers’  results  provided  reasonable  verification  of  coding  procedure  accuracy  (Creswell, 
2003;  Isabella,  1990).  The  patterns  and  themes  revealed  through  this  coding  are  described 
below  in  greater  detail  for  each  interview  question. 

Suggested  Metrics  for  “Focus  on  results,  not  process”  Initiative  Implementation. 

Two  interview  questions  directly  addressed  metrics  for  the  initiative  of  interest.  Interview 
participants  were  asked  what  metrics  they  would  use  to  measure  the  results  the  initiative  was 
meant  to  bring  about,  and  then  later  in  the  interview  participants  were  asked  how  they  would 
know  if  they  were  succeeding  at  implementing  the  initiative.  Out  of  participants’  responses,  five 
main  categories  of  metrics  were  recurrent:  schedule,  customer  satisfaction,  cost,  performance, 
and  credibility.  The  remaining  interview  excerpts  for  these  questions  that  did  not  seem  to  fit  into 
a  particular  category  were  placed  in  a  miscellaneous  category. 

The  theme  most  identified  by  participants  was  schedule,  or  acquisition  program 
baseline,  which  refers  to  the  lengths  of  time  a  program  has  set  to  accomplish  various  tasks. 

This  category,  which  relates  directly  to  Sambur’s  (Senate  Armed  Services  Committee,  2002) 
call  for  improved  speed,  also  included  a  sub-category  of  cycle  time,  meaning  the  length  of  time 
from  identifying  a  need  for  something  until  it  is  delivered.  Cycle  time,  in  turn,  included  two 
subcategories.  Capability  based  cycle  time  refers  to  the  amount  of  time  between  the  warfighter 
stating  his  need  for  a  new  capability  and  that  capability  being  delivered.  Documentation  based 
cycle  time  refers  to  acquisition  lead-time  or  the  time  it  takes  to  complete  a  document  related 
activity,  e.g.,  contract  negotiation  and  award.  Seventy  four  percent  of  participants  identified 
schedule  as  a  metric  category  for  the  initiative  of  interest. 

Customer  satisfaction  with  the  product,  process,  or  service  being  provided  was  the 
second  most  frequently  named  metric  category.  In  the  participants’  view,  customer  satisfaction 
also  encompasses  a  sub-category  of  expectation  management.  A  large  part  of  how  satisfied 
the  customer  is  depends  on  whether  they  received  what  they  were  expecting.  Sixty  one  percent 
of  participants  named  customer  satisfaction  as  a  metric  category. 

Cost  and  performance  were  the  third  and  forth  most  identified  themes.  Cost  is  self 
explanatory;  it  deals  with  activities  related  to  money.  This  theme  occurred  among  thirty  nine 
percent  of  participants.  Performance,  how  well  a  program  performs  to  specifications  and 
expectations,  was  the  fourth  most  recurring  theme  for  these  interview  questions.  Both  customer 
satisfaction  and  performance  address  expectations;  of  the  two,  performance  is  the  more  direct 
comparison  against  expected  capability,  and  customer  satisfaction  addresses  a  more 
comprehensive  assessment  of  all  customer  expectations.  Twenty  six  percent  of  participants 
suggested  metrics  that  fit  into  this  category. 

Credibility  was  the  fifth  category  of  participant  interview  responses.  Credibility  for  the 
acquisition  workforce  would  mean  that  their  customers,  mainly  the  warfighters,  would  believe 
what  they  tell  them  and  find  them  trustworthy.  This  also  ties  in  with  Sambur’s  (Senate  Armed 
Services  Committee,  2002)  call  to  improve  credibility;  the  acquisition  community  has  to  deliver 
what  they  say  they  are  going  to  deliver.  However,  the  occurrence  of  this  theme  among  only 
seventeen  percent  of  participants  does  not  seem  to  support  Sambur’s  push  for  improved 
credibility;  this  indicates  that  not  many  people  see  measuring  credibility  as  a  way  of  telling  if  this 
initiative  is  succeeding.  In  contrast,  the  frequent  recurrence  of  schedule  among  seventy  four 
percent  of  participants  does  offer  support  for  Sambur’s  call  for  speed. 

Meaning  of  2002  Lighting  Bolt  “Focus  on  results,  not  process”  Initiative. 
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Interview  participants  were  asked  what  they  thought  the  initiative  of  interest  meant. 
Interview  participants  included  individuals  who  helped  to  draft  the  initiative,  one  of  whom  stated 
the  following  about  what  the  initiative  was  intended  to  mean: 

Too  many  people  within  the  acquisition  community  focus  on  completing  processes 
(reports,  assessments,  checklists,  etc).  The  Lighting  Bolt  aimed  to  cause  people  to  look 
at  the  result  intended  by  the  process  and  to  make  a  judgement  of  whether  the  activity 
planned  actually  furthers  the  opportunity  for  success.  Success  isn’t  getting  through  the 
process  -  its  delivering  a  needed  capability  to  the  warfighter! 

A  variety  of  other  responses  were  provided  by  participants  and  then  grouped  by  the 
themes  that  emerged.  The  top  five  responses  were: 

•  Focus  on  the  end  customer  not  the  acquisition  process  itself;  support  the  customer 

•  Does  what  we  are  doing  make  sense  and  does  it  add  value?  If  not,  get  rid  of  it  or 
waive  it;  remove  the  unnecessary  steps 

•  Freeing  people  up 

•  Focus  on  getting  the  product  out;  effects  based  or  outcome  based  acquisitions 

•  Want  results  not  just  process 

Desired  Results  or  Outcomes  of  Initiative. 

Next,  participants  were  asked  what  they  thought  were  the  desired  outcomes  or  results 
that  the  initiative  was  trying  to  accomplish.  Multiple  themes  were  identified  from  the  interview 
data,  the  first  of  which  occurred  within  eleven  excerpts  among  nine  respondents,  the  second 
from  ten  excerpts  between  eight  respondents,  and  the  remainder  from  five  excerpts  among  four 
respondents  or  less.  The  top  five  responses  were: 

•  Support  the  Agile  Acquisition  strategy;  provide  capability  in  a  timely  way  without 
getting  bogged  down  in  the  processes 

•  Change  people’s  way  of  thinking;  be  creative,  innovative,  and  use  common  sense 

•  Get  people  to  think  about  the  outcome  not  the  how 

•  Promises  made,  promises  kept 

•  Roadblocks  exist  to  accomplishing  initiative  outcomes  from  1)  middle  management, 
2)  SAF/AQ  staff  and  other  services,  and  3)  contracting 

Appropriateness  of  Initiative  Goals. 

Participants  were  also  asked  if  they  thought  the  goals  of  the  initiative  were  appropriate. 
Based  on  their  understanding  of  what  they  thought  the  goals  or  outcomes  of  the  initiative  to  be, 
twenty  of  the  twenty  three  participants  agreed  the  goals  were  appropriate.  However,  when 
asked  if  they  felt  using  this  initiative  was  the  most  appropriate  way  to  accomplish  the  goals  that 
it  was  meant  to  accomplish,  several  participants  offered  various  criticisms  of  the  initiative, 
including  inadequate  education  about  its  motive,  a  perception  that  the  initiative  was  successful 
only  while  its  champion  ran  Air  Force  acquisition,  and  a  general  lack  of  top  management 
support  beyond  the  champion. 

Most  Important  Aspects  of  the  Initiative. 

The  interview  participants  were  then  asked  what  they  considered  to  be  the  most 
important  aspects  of  the  “Focus  on  results,  not  process”  initiative.  Four  main  themes  were 
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identified  within  the  data  and  were  close  in  frequency  of  occurrence  among  participants. 
Organizational  culture  was  the  most  recurring  theme;  nine  excerpts  from  six  participants 
reflected  this  theme.  Participants  stated  that  an  entrepreneurial  mindset  was  the  next  most 
important  aspect  which  includes,  but  is  not  limited  to,  becoming  creative,  not  being  risk  averse, 
taking  bold  steps  to  challenge  the  status  quo,  and  thinking  differently.  Seven  interview  excerpts 
among  seven  participants  noted  this  aspect.  Responsiveness  to  the  customer  was  the  next 
most  frequently  seen  theme  with  six  excerpts  among  five  participants.  Lastly,  five  excerpts  from 
four  participants  shared  the  theme  of  communication. 

How  Participants  Heard  about  the  initiative. 

Next  participants  were  asked  how  they  had  actually  heard  of  the  initiative.  Four 
participants  said  that  they  had  not  heard  of  the  “Focus  on  results,  not  process”  concept  as  a 
formal  initiative  until  they  were  contacted  about  this  research  project;  but,  based  on  their 
interview  responses  they  had  actually  already  been  carrying  out  the  intent  of  the  initiative  within 
their  jobs.  Those  participants  included  two  SPO  program  managers,  a  Deputy  SPO  Director, 
and  an  Air  Staff  member.  Among  those  participants  who  had  heard  of  the  initiative,  the  sources 
from  which  they  learned  of  the  initiative  were  varied.  The  most  frequent  source  of  introduction 
to  the  initiative  was  through  participants’  chains  of  command  and  normal  information  distribution 
channels;  seven  excerpts  from  seven  participants  shared  this  theme.  The  next  most  recurring 
theme  was  direct  involvement  with  Darleen  Druyun,  the  originator  of  the  initiative.  Six  excerpts 
from  five  participants  shared  this  theme.  Three  excerpts  from  three  participants  noted  direct 
contributions  to  writing  the  initiative.  And  the  three  remaining  themes  observed  from  single 
excerpts  among  individual  participants  were  acquisition  reform  training,  Sambur’s  (Department 
of  the  Air  Force,  4  February  2003)  letter  to  the  acquisition  community  introducing  the  initiative  as 
part  of  the  new  push  for  improved  speed  and  credibility,  and  working  in  an  Acquisition  Center  of 
Excellence  (ACE)  office. 

Next  Step  in  “Focusing  on  Results”. 

In  addition  to  being  asked  to  explain  how  they  had  heard  about  the  initiative,  participants 
were  asked  what  the  next  step  should  be  in  order  to  get  the  acquisition  community  to  actually 
implement  the  objectives  of  the  initiative  and  really  focus  on  the  results.  There  were  as  many 
responses  to  this  question  as  there  were  interview  participants.  The  most  frequent  theme  within 
the  interview  responses  for  this  question  was  seen  within  six  excerpts  shared  among  four 
participants.  The  second  most  frequent  theme  came  from  six  excerpts  among  three 
participants.  The  next  two  most  frequent  themes  were  seen  in  three  excerpts  from  three 
participants.  All  of  the  other  themes  were  shared  by  only  two  participants  or  less.  The  top  four 
responses  were: 

•  Change  the  acquisition  workforce  culture 

•  Apply  the  initiative  to  the  processes  that  support  Evolutionary  Acquisition 

•  Training  and  education 

•  Expectation  management  with  the  warfighters  and  Air  Staff,  and  in  turn  Congress 
Organization  Implementation  of  the  Initiative. 

After  stating  what  they  thought  would  be  the  next  steps  to  take  in  order  to  get  people  to 
accomplish  the  goals  of  the  initiative,  participants  were  asked  how  their  organizations  were 
implementing  the  initiative.  The  themes  from  the  data  describing  organizational  use  of  the 
initiative  are  listed  below.  The  most  frequent  theme  incorporates  seven  excerpts  from  five 
participants  who  are  members  of  various  Acquisition  Center  of  Excellence  (ACE)  offices;  the 
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second  most  frequent  theme  was  from  seven  excerpts  among  three  participants;  and  the  third 
most  frequently  occurring  theme  was  shared  by  four  excerpts  from  four  participants.  These  top 
three  themes  were 

•  ACE  offices  assist  programs  to  challenge  burdensome  processes  and  try  to  influence 
people  to  use  the  philosophy  of  the  initiative 

•  Stress  full  participation  of  Integrated  Product  Teams  (IPTs)  and  ensure  IPTs  include 
the  warfighters,  contractors,  and  contracting  officers 

•  Rewrote  Air  Force  Instructions  and  Air  Force  Federal  Acquisition  Regulation 
Supplements  and  other  guiding  documents  to  free  up  people 

Most  participants  indicated  they  were  implementing  the  initiative  in  some  way,  but 
several  said  they  were  not.  Most  notably,  several  middle  management  participants  said  they 
were  not  actively  implementing  the  initiative  because  from  their  perspectives  Air  Force 
acquisition  and  sustainment  were  process  oriented  than  product  oriented.  They  referred  to 
required  participation  in  lecture  series  and  workshops  by  renowned  process  re-engineering 
advocate  Michael  Hammer  (Hammer,  2001  :i).  Another  participant  stated  that  they  would  not 
remove  non-value  added  acquisition  processes  because  they  would  not  challenge  the  Federal 
Acquisition  Regulation  without  a  lawyer.  And  one  other  participant  said  they  were  having 
difficulty  implementing  the  initiative. 

Participants’  Roles  in  the  Development  or  Implementation  of  the  Initiative. 

Not  only  were  participants  asked  how  their  organizations  were  implementing  the 
initiative,  each  participant  was  asked  what  his  or  her  specific  role  was  in  either  the  development 
or  implementation  of  the  initiative.  The  most  recurring  theme,  from  fourteen  excerpts  among 
eight  participants,  was  that  participants  acted  as  enablers  for  their  teams  by  challenging  their 
teams  to  use  the  initiative;  running  interference  for  their  teams  when  their  attempts  to  implement 
the  initiative  met  resistance;  and  developing  and  maintaining  good  relationships  with  people 
involved  with  the  acquisition.  All  other  themes  came  from  three  excerpts  from  three  participants 
or  less.  The  top  five  themes  were: 

•  Being  an  enabler  for  your  team 

•  Developer  or  author  of  the  initiative 

•  Endorser  and  advocate  of  the  initiative 

•  Had  no  role  in  the  development  of  the  initiative 

•  Provide  advice  to  senior  leadership  on  ways  to  implement  the  initiative 

Support  for  Organizational  Implementation  of  Initiative. 

After  participants  were  asked  about  their  roles  in  the  development  and  implementation  of 
the  initiative,  they  were  asked  several  questions  about  the  level  of  support  they  are  receiving  in 
their  attempts  to  implement  the  initiative.  The  first  of  these  questions  asked  what  kind  of 
support  participants’  organizations  were  getting  as  they  try  to  use  the  initiative.  The  most 
recurring  theme  that  was  seen  in  responses  to  this  question  came  from  eighteen  excerpts  given 
by  thirteen  participants.  The  other  themes  were  expressed  in  three  excerpts  among  three 
participants  or  less.  The  top  five  themes  were: 

•  Top  down  support 

•  ACE  help  in  planning  for  program  events 

•  Contractor  support 
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•  Initiative  training;  risk  management  training  and  Discovery  Map  training 

•  Being  left  alone  and  trusted  to  go  implement  the  initiative  is  the  best  support 

However,  several  negative  themes  about  the  level  of  support  organizations  were 
receiving  arose  from  the  responses  of  seven  participants. 

•  No  support  is  being  given 

•  The  bureaucracy  is  fighting  implementation  of  the  initiative 

•  Senior  leaders  empowered  the  workforce  to  go  out  and  implement  the  initiative,  but 
they  are  not  preaching  it  enough  themselves;  need  strong,  consistent  advocacy 

•  Headquarters  puts  the  initiatives  out  but  does  not  have  to  live  with  them 

Support  for  Individual  Implementation  of  Initiative. 

The  next  support  related  question  dealt  with  whether  or  not  they  felt  they  were  getting 
the  support  they  needed  to  implement  the  initiative.  Over  sixty  five  percent  of  the  participants 
said  they  were  receiving  the  support  they  needed  to  implement  the  initiative  from  those  within 
their  chain  of  command  and  from  those  areas  within  their  control.  However,  thirty  percent  of 
participants  said  they  were  not  getting  the  support  they  needed  from  those  who  are  outside  of 
their  chain  of  command  but  can  still  influence  their  ability  to  implement  the  initiative.  Five 
percent  of  participants  were  undecided.  Participants  noted  that  they  were  not  getting  support 
from  Headquarters  Air  Force  (HQ  USAF),  Air  Staff,  or  the  Office  of  the  Secretary  of  Defense. 
The  need  for  consistent,  repeated,  vocal  support  from  senior  Air  Force  leaders  and  the  need  for 
buy-in  from  people  and  processes  outside  of  the  immediate  Air  Force  chain  of  command  were 
recurring  themes  among  the  participants. 

Organizational  Support  for  Individual  Implementation  of  Initiative. 

Participants  were  also  asked  how  their  organizations  were  supporting  them  in  their 
attempts  to  put  the  initiative  into  action.  The  theme  of  support  and  encouragement  being 
provided  by  leadership  within  participants’  direct  chains  of  command  was  noted  among  twelve 
excerpts  from  eleven  out  of  twenty  three  participants.  Three  other  participants,  including  two 
middle  management  members,  said  they  received  support  from  their  organizations  by  being 
trusted  to  do  the  job  and  being  left  alone  to  do  it.  Three  excerpts  from  one  participant  called  out 
strong  support  from  the  ACE  offices  as  an  avenue  of  organizational  support.  Another 
participant  said  they  were  getting  support  from  their  organization  by  virtue  of  having  no  kick- 
back  from  SPO  members  which  indicated  that  the  SPO  members  have  accepted  the  challenge 
for  their  organization  to  implement  the  initiative.  The  final  theme  that  arose  out  of  excerpts  for 
this  question  was  from  a  participant  who  twice  stated  that  they  were  not  seeing  leaders  at  the 
Senior  Executive  Service  (SES)  and  General  Officer  level  engage  enough  in  the  drive  to  use 
this  initiative;  the  participant  considers  leaders’  involvement  to  be  one  of  the  most  important 
tools  they  need  to  do  their  job. 

Initiative  Implementation  Success  Stories. 
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After  participants  were  asked  about  the  level  of  support  they  were  receiving  in  their 
attempts  to  utilize  the  initiative,  they  were  asked  if  they  had  heard  of  any  success  stories  or 
failures  at  using  the  initiative.  Eleven  of  the  twenty  three  participants  said  they  could  cite  no 
specific  examples  of  success  stories,  but  eleven  other  participants  did  provide  examples  of  what 
they  considered  to  be  successes.  The  success  stories  were  grouped  into  two  categories: 
process  level  successes  and  program  level  successes. 

Process  level  successes  are  ways  the  initiative  of  interest  has  been  used  to  remove  non 
value-added  processes  and  which  can  be  repeated  within  program  offices  across  the  entire  Air 
Force.  Participants  cited  several  examples,  such  as  including  a  source  selection  plan  in  a 
System  Acquisition  Management  Plan  in  order  to  get  approval  for  both  at  the  same  time; 
incorporating  a  Price  Competition  Memo  (PCM)  in  a  Proposal  Analysis  Report  (PAR)  which 
reduced  time  because  now  the  same  pricing  structure  can  be  used  for  both  the  PAR  and  PCM; 
successfully  challenging  the  need  for  Mission  Need  Statement  (MNS)  and  Operational 
Requirements  Document  (ORD)  on  a  common  computer  purchase  and  saving  an  estimated  six 
months  of  work  to  do  the  MNS  and  ORD;  delegation  of  contract  approval  authority  down  to 
various  base  level  personnel,  so  that  people  now  rarely  have  to  go  to  higher  headquarters  for 
approvals. 

In  addition  to  process  level  success  stories,  several  weapon  system  programs  were 
recognized  by  participants  as  examples  of  how  the  initiative  can  be  utilized  successfully.  One 
such  program  is  the  Crystal  Modification  Program.  The  organization  running  that  program  was 
able  to  go  and  influence  the  Army  and  Navy  to  combine  functionalities  of  boxes  where  it  made 
sense  to  do  so  and  reduce  the  footprint,  and,  in  turn,  reduce  the  development  costs  and 
infrastructure  costs.  Several  other  programs  were  also  identified  as  success  stories  because  of 
how  they  kept  their  focus  on  the  results  being  delivered  to  the  warfighters  and  how  they  did  not 
get  bogged  down  in  the  acquisition  process  itself.  Programs  like  Global  Hawk,  Micro  Impulse 
Radar,  Patient  Support  Pallet,  and  the  weaponization  of  Predator  are  additional  examples  that 
participants  considered  success  stories  of  how  the  initiative  of  interest  can  be  used. 

Initiative  Implementation  Failures. 

Forty  three  percent  of  interview  participants  indicated  that  they  had  not  heard  of  any 
specific  examples  of  failure  at  using  the  initiative,  though  several  others  did  identify  process- 
level  failures.  One  participant  discussed  how  the  Air  Force’s  process  for  reprogramming  funds, 
which  allows  money  to  be  used  for  programs  other  than  what  it  was  originally  slated  for,  does 
not  enact  the  initiative.  The  impression  among  the  Air  Force  workforce  is  that  Congress  is  why 
it  takes  too  long  to  approve  reprogramming.  The  participant’s  office  checked  and  found  out  that 
Congress  only  takes  thirty  of  the  hundred  and  fifty-eight  day  cycle  to  reprogram  funding;  the  rest 
is  taken  up  by  the  Air  Force.  So  if  money  has  to  be  reprogrammed  above  certain  approval 
thresholds,  it  takes  an  average  of  a  hundred  and  thirty  days  just  to  process  the  request  through 
the  Pentagon. 

Another  process  level  failure  example  involved  an  attempt  to  do  a  zero  baseline  of  all 
work  in  a  SPO  at  Electronic  Systems  Center.  This  exercise  was  originally  designed  to 
challenge  the  value-added  contribution  of  every  activity  that  the  program  office  was  doing  by 
forcing  each  activity  and  report  to  justify  its  contribution.  SPO  personnel  were  not  interested  in 
doing  it.  This  was  clearly  a  failure  at  implementing  the  initiative.  After  participants  were  asked 
about  successes  and  failures  at  using  the  initiative  of  interest,  they  were  asked  questions  about 
acquisition  reform  initiatives  in  general. 
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Being  Successful  at  Implementing  Any  Acquisition  Reform  Initiative. 

The  last  two  interview  questions  were  applicable  to  acquisition  reform  initiatives  on  a 
broader  scale.  The  first  of  these  two  questions  asked  participants  what  they  think  it  takes  to  be 
successful  at  utilizing  any  acquisition  reform  initiative.  A  range  of  themes  emerged  from  their 
responses.  The  most  frequently  occurring  theme  was  seen  among  eleven  interview  excerpts 
from  seven  participants.  The  second  most  frequent  theme  came  from  seven  excerpts  among 
seven  participants.  The  third  most  frequent  theme  was  common  among  five  excerpts  from  five 
participants.  And  the  fourth  most  common  theme  was  from  six  excerpts  among  four 
participants.  The  top  four  themes  were: 

•  Consistent  message  from  the  top 

•  Senior  leadership  buy-in;  support  and  advocacy  for  initiative  from  senior  leaders 

•  Be  very  specific  in  what  the  initiative  says  and  in  what  is  expected  of  those  who  use 
it 

•  Behavior  of  leaders  has  to  reinforce  philosophy  behind  the  initiative 

Important  Elements  of  an  Acquisition  Reform  Initiative. 

Lastly,  after  interview  participants  were  asked  about  what  they  considered  to  be  the  keys 
to  successful  initiative  implementation,  they  were  asked  what  was  important  to  them  in  any 
acquisition  reform  initiative.  Many  of  the  themes  that  emerged  from  the  data  mirrored  the 
characteristics  of  good  metrics  found  within  literature.  The  most  recurring  theme  was  from  six 
interview  excerpts  among  five  participants.  The  next  three  most  recurring  themes  were  each 
common  among  four  excerpts  from  four  participants.  The  other  themes  produced  came  from 
five  excerpts  from  three  participants  or  less.  The  top  four  themes  were: 

•  Focus  on  the  mission  of  the  Air  Force  and  getting  something  to  the  warfighters 

•  Be  beneficial  to  the  acquisition  grunts  and  the  end  users 

•  Makes  sense 

•  Follow  through;  see  it  through  to  the  end 

Summary 

After  the  interview  data  were  carefully  examined  using  methods  modeled  after  Creswell 
(2003)  and  Isabella  (1990),  excerpts  from  participants’  responses  that  represented  the  core  of 
their  answers  to  each  question  were  grouped  by  the  themes  that  emerged.  Those  themes 
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revealed  participants’  opinions  about  the  kinds  of  metrics  they  would  use  to  measure  the  “Focus 
on  results,  not  process”  initiative’s  success,  the  meaning  and  goals  of  the  initiative,  the  next  step 
in  achieving  the  initiative’s  goals,  how  they  heard  about  it,  the  kind  of  support  initiative 
implementation  is  receiving,  successes  and  failures  at  initiative  utilization,  and  how  to  make 
generic  initiatives  successful.  This  analysis  will  serve  as  the  basis  for  several  conclusions  and 
recommendations  discussed  in  the  final  section  of  the  paper. 

Conclusions  &  Recommendations 

This  research  has  attempted  to  help  senior  Air  Force  leaders  build  a  framework  for 
developing  measures  of  success  for  corporate  level  Air  Force  acquisition  initiatives.  This 
section  discusses  the  conclusions  and  recommendations  based  on  the  analysis. 

Conclusions 

Five  conclusions  were  drawn  from  the  interview  data.  First,  a  breakdown  in 
communication  about  what  the  initiative  meant  occurred  throughout  the  acquisition  workforce; 
no  consistent  definition  for  the  initiative  was  found  among  participants  except  among  the 
participants  who  helped  author  the  initiative  and  those  who  work  in  ACE  offices.  Many  people 
took  the  initiative  to  mean  “if  the  acquisition  process  in  question  was  not  a  law,  then  break  it”. 
The  authors  of  the  initiative,  however,  stated  the  intent  was  for  people  to  challenge  non-value 
added  processes  with  well-supported  waiver  requests. 

Secondly,  disconnects  exist  between  the  middle  management  level  and  the  other 
management  levels  on  several  fronts.  Middle  management  shared  a  unanimous  view  on  what 
the  initiative  was  intended  to  accomplish,  but  that  view  differed  from  the  view  that  senior  leaders 
and  SPO  level  workers  shared.  Senior  leaders  provided  guidance  on  how  to  apply  the  initiative 
and  on  what  the  initiative  means  in  the  form  of  policy  letters,  directives,  and  briefings 
(Department  of  the  Air  Force,  4  February  2003;  Department  of  the  Air  Force,  10  July  2003; 
Senate  Armed  Services  Committee,  2002).  In  addition,  SPO  level  workers  provided  numerous 
examples  of  success  stories  at  implementing  the  initiative  within  their  program  offices,  showing 
a  clear  understanding  of  the  initiative  and  active  use  of  it.  However,  middle  management  thinks 
the  initiative  is  a  “dead  horse,”  and  cited  a  countervailing  emphasis  on  process  reform 
(“Hammer  training”)  as  confusing  the  focus.  Finally,  a  possible  confound  is  that  middle 
management’s  disconnects  with  the  other  management  levels  may  be  due  in  part  to  some  bias 
towards  the  initiative  originator. 

The  third  conclusion  is  that  participants  perceive  that  administrative  hurdles  to 
implementing  the  initiative  are  being  put  up  by  organizations  and  agencies  outside  of 
participants’  chains  of  command.  Participants  clearly  stated  that  they  are  getting  the  support 
they  need  from  their  immediate  bosses,  but  that  there  are  obstacles  from  outside  organizations 
and  agencies;  for  example,  added  oversight  from  Congress,  having  to  work  with  the  Department 
of  Defense  and  other  services,  and  having  to  get  approvals  from  people  outside  of  their  decision 
chain  of  command.  Thirty  percent  of  participants  stated  that  those  outside  their  chain  of 
command,  who  can  still  influence  their  level  of  success  at  implementing  the  initiative,  are  not 
providing  the  support  participants  need.  Numerous  participants  also  included  the  Office  of  the 
Secretary  Of  Defense  (OSD)  among  their  biggest  perceived  roadblocks. 

A  fourth  conclusion  was  that  differing  perceptions  exist  about  to  whom  the  initiative 
applies.  According  to  some  participants,  the  dubbing  of  the  initiative  as  an  acquisition  reform 
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initiative  led  people  within  the  requirements  arena  (e.g.,  Air  Combat  Command),  people  in  the 
testing  community,  and  those  in  the  logistics  and  weapon  system  sustainment  community  to 
think  the  initiative  was  only  geared  towards  weapon  system  acquisition  offices.  The  negative 
responses  from  many  of  the  people  approached  to  participate  in  this  research  cited  the  point 
that  they  did  not  think  the  initiative  applied  to  them  as  the  reason  they  could  not  help  with  this 
research;  Test  and  Evaluation  Centers,  Air  Logistics  Centers,  and  an  Air  Force  Space 
Command  System  Program  Office  (SPO)  did  not  participate  in  this  research  for  that  reason. 

The  final  conclusion  drawn  is  that  using  an  initiative  like  this  may  not  be  the  most 
effective  way  to  accomplish  the  desired  behavioral  change.  Just  sending  out  an  initiative  and 
leaving  it  up  the  workers  in  the  acquisition  trenches  to  figure  out  how  to  apply  it  will  not  cut  it. 
The  need  for  leadership  and  people’s  fears  of  change  and  failure  need  to  be  addressed. 
Participants,  including  the  middle  management  personnel,  consistently  said  that  senior  leaders 
need  to  be  more  engaged  and  regularly  vocal  about  the  importance  of  things  like  the  initiative 
for  it  to  be  successful.  After  conclusions  were  reached,  recommendations  were  developed 
using  both  the  data  analysis  and  literature  review  findings. 

Recommendations 

These  five  conclusions  suggest  several  recommendations.  The  first  recommendation  of 
this  research  will  only  address  the  metric  category  of  customer  satisfaction  because  the 
categories  of  schedule,  cost,  credibility,  and  performance  have  been  previously  addressed  by 
other  Air  Force  agencies  and  because  cost,  schedule,  and  performance  metrics  are  already 
broadly  used  across  the  Air  Force  (Air  Force  Inspection  Agency,  2003:84). 

This  research  recommends  the  use  of  customer  satisfaction  metrics  to  measure  the 
success  of  the  “Focus  on  results,  not  process”  initiative.  Customer  satisfaction  is  described  as 
the  extent  to  which  a  process  or  product  meets  a  customer’s  expectations  (Kotler  and 
Armstrong,  2001;  Naumann  and  Jackson,  1999;  Zeithaml  and  Bitner,  2006).  Customer 
satisfaction  is  the  key  to  organizational  success  (Gibson  et  al.,  2003).  No  matter  how  precisely 
a  schedule  is  maintained,  how  much  cost  savings  are  realized,  how  credible  the  end  customer 
thinks  the  acquisition  community  is,  or  what  exceptional  performance  a  weapon  system  or 
process  has,  if  the  customer  is  not  satisfied  with  the  result,  the  acquisition  community  has  failed. 
While  multiple  customer  satisfaction  metrics  could  apply  and  one  size  does  not  fit  all  (INCOSE, 
1998:9),  the  following  suggested  metrics  could  prove  very  useful. 

A  suggested  metric  for  schedule  is  timeliness  (Ellis  and  Curtis,  1995;  Hayes,  1992). 
Suggested  metrics  for  performance  are  reliability  and  perceived  quality  (Ellis  and  Curtis,  1995; 
Naumann  and  Jackson,  1999).  And  a  suggested  metric  for  the  area  of  credibility  is 
responsiveness  (Ellis  and  Curtis,  1995;  Hayes,  1992;  Naumann  and  Jackson,  1999).  A 
customer  satisfaction  metric  for  cost  is  not  suggested  because,  according  to  Hammer  (2001),  it 
tells  very  little  if  anything  about  the  business.  The  list  of  metrics  is  general  since  the  attributes 
of  each  dimension  are  very  product  specific,  meaning  the  metrics  should  be  tailored  for  a  better 
fit  depending  upon  what  product  or  process  they  are  applied  to  (Naumann  and  Jackson,  1999). 

Expand  Innovation  Education  for  the  Acquisition  Workforce. 

The  acquisition  workforce  has  traditionally  been  trained  in  how  to  use  the  Federal 
Acquisition  Regulation  (FAR)  and  other  guides  and  instructions;  but,  if  members  of  the 
acquisition  community  are  now  expected  to  be  innovative,  they  have  to  be  trained  in  how  to  do 
that.  One  middle  management  participant  related  this  anecdote  from  a  member  of  an  audience 
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to  which  he  had  spoken,  “I  used  to  be  able  to  sit  down  at  my  desk  and  open  my  cookbook  and 
follow  the  recipe,  and  I'd  get  done  with  the  product.  You  took  my  recipe  book  away  from  me, 
and  I  don't  know  what  to  do.”  One  avenue  to  train  the  acquisition  community  is  to  expand 
educational  efforts  in  areas  such  as  strategic  purchasing,  the  entrepreneurial  mindset,  and 
organizational  management  and  transformation. 

Continually  Challenge  Waiver  Processes  to  Reduce  Approval  Effort. 

A  pervasive  opinion  among  respondents  was  that  requesting  waivers  to  existing 
processes,  however  broken  they  might  be  perceived  to  be,  is  too  time  intensive.  Waiver 
processes  should  be  subject  to  continual  review  to  ensure  they  support  change  and  not 
discourage  it.  This  view  is  supported  by  one  senior  respondent,  who  said: 

“So  what's  important  for  me  in  an  initiative  is  that  it  be  something  I  can  do  and  it  would 
be  value  added  and  it  would  actually  cut  my  work.  In  other  words,  don't  tell  me...  I  can 
have  an  exception  to  somethin'  if  I  have  to  go  ask  for  --  you  know,  I  have  to  go  sell  this 
exception  to  every  layer  of  bureaucracy  I've  gotta  work  with.  It  just  doesn't--  it's--  it's  just 
here,  let  me  do  it  the  regular  way.” 

Clarify  Future  Initiatives. 

One  middle  management  participant  noted  that  when  the  term  acquisition  reform 
initiative  is  used  “the  rest  of  the  Air  Force  thinks  it  only  applies  to  the  acquisition  world.”  Future 
initiatives  should  not  use  buzzwords  like  acquisition  or  acquisition  reform,  but  instead  appeal  to 
the  Air  Force  as  a  whole.  The  new  Air  Force  initiative  “Smart  Ops  21”,  which  focuses  on 
process  improvement  across  the  Air  Force,  is  an  encouraging  step  in  this  direction.  Future 
initiatives  should  be  very  specific,  and  metrics  and  a  commander’s  intent  statement  should  be 
released  along  with  the  initiatives.  The  initial  metrics  can  be  adjusted  if  they  prove  to  be  driving 
undesirable  behaviors,  but  this  study  suggests  starting  with  some  metrics  is  better  than  starting 
with  none. 

Consistent  Statement  of  Initiative  Support  from  Senior  Air  Force  Leaders. 

Lastly,  it  is  recommended  that  both  the  Secretary  of  the  Air  Force  and  the  Chief  of  Staff 
of  the  Air  Force  be  asked  to  include  periodic  statements  of  support  for  continued  acquisition 
reform,  stressing  the  importance  of  compliance  with  acquisition  initiatives  in  their  monthly 
newsletters  (Vectors  and  Sight  Pictures,  respectively).  Such  statements  would  address  the 
prevailing  and  pervasive  request  from  interview  participants  and  others  for  consistent,  continual, 
vocal  support  for  the  initiative  of  interest  from  the  Chief  and  other  senior  Air  Force  leaders.  The 
Chief  and  the  Secretary’s  continued  and  open  support  would  enable  those  expected  to 
implement  the  initiatives  to  really  challenge  the  party-line  way  of  doing  acquisitions,  become 
innovative,  and  change  the  acquisition  process. 

Future  Research 

The  future  research  should  focus  first  on  validating  the  findings  of  the  qualitative 
research.  A  questionnaire  can  be  constructed  using  the  data  gathered  from  the  interviews. 

The  questionnaire  can  be  used  to  evaluate  the  generated  metrics  along  the  dimensions  of 
“good”  metrics;  specific  metrics  that  apply  to  the  categories  of  metrics  this  research  will  suggest 
can  then  be  identified  during  future  research. 
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Summary 


This  study  gives  Air  Force  leaders  clear,  implementable  metrics  that  can  be  used  as 
measures  of  success  for  the  “Focus  on  results,  not  process”  initiative,  and  provides 
recommendations  that  can  be  used  to  improve  this  initiative’s  performance  and  that  of  future 
corporate  Air  Force  acquisition  initiatives.  This  study  also  gives  Air  Force  leaders  insight  into 
whether  or  not  this  initiative  and  others  like  it  are  an  appropriate  and  effective  way  to  drive  the 
changes  they  are  meant  to  bring  about.  Finally,  from  a  broader  perspective,  the  framework 
used  in  this  study  can  be  used  to  develop  measures  of  success  for  other  corporate  level  Air 
Force  acquisition  initiatives. 
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2003  -  2006  Sponsored  Acquisition  Research  Topics 

Acquisition  Management 

■  Software  Requirements  for  OA 

■  Managing  Services  Supply  Chain 

■  Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

■  Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to  Shipyard 
Planning  Processes 

■  Portfolio  Optimization  via  KVA  +  RO 

■  MOSA  Contracting  Implications 

■  Strategy  for  Defense  Acquisition  Research 

■  Spiral  Development 

■  BCA:  Contractor  vs.  Organic  Growth 
Contract  Management 

■  USAF  IT  Commodity  Council 

■  Contractors  in  21st  Century  Combat  Zone 

■  Joint  Contingency  Contracting 

■  Navy  Contract  Writing  Guide 

■  Commodity  Sourcing  Strategies 

■  Past  Performance  in  Source  Selection 

■  USMC  Contingency  Contracting 

■  Transforming  DoD  Contract  Closeout 

■  Model  for  Optimizing  Contingency  Contracting  Planning  and  Execution 
Financial  Management 

■  PPPs  and  Government  Financing 

■  Energy  Saving  Contracts/DoD  Mobile  Assets 

■  Capital  Budgeting  for  DoD 

■  Financing  DoD  Budget  via  PPPs 

■  ROI  of  Information  Warfare  Systems 

■  Acquisitions  via  leasing:  MPS  case 

■  Special  Termination  Liability  in  MDAPs 
Logistics  Management 

■  R-TOC  Aegis  Microwave  Power  Tubes 
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■  Privatization-NOSL/NAWCI 

■  Army  LOG  MOD 

-  PBL  (4) 

■  Contractors  Supporting  Military  Operations 

•  RFID  (4) 

■  Strategic  Sourcing 

■  ASDS  Product  Support  Analysis 

■  Analysis  of  LAV  Depot  Maintenance 

■  Diffusion/Variability  on  Vendor  Performance  Evaluation 

■  Optimizing  CIWS  Life  Cycle  Support  (LCS) 

Program  Management 

■  Building  Collaborative  Capacity 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  Aegis  and  SSDS 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module 
Acquisition 

■  Terminating  Your  Own  Program 

■  Collaborative  IT  Tools  Leveraging  Competence 

A  complete  listing  and  electronic  copies  of  published  research  within  the  Acquisition 
Research  Program  are  available  on  our  website:  www.acquisitionresearch.org 
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